Skip to content

feat: central graph store (graphify remote init/push/pull/delete) — graphify-out links into a store folder + repo-local sync hooks#1752

Open
muthuishere wants to merge 2 commits into
Graphify-Labs:v8from
muthuishere:feat/config-store-and-sync-hooks
Open

feat: central graph store (graphify remote init/push/pull/delete) — graphify-out links into a store folder + repo-local sync hooks#1752
muthuishere wants to merge 2 commits into
Graphify-Labs:v8from
muthuishere:feat/config-store-and-sync-hooks

Conversation

@muthuishere

@muthuishere muthuishere commented Jul 9, 2026

Copy link
Copy Markdown

What

Teams with large graphs (hundreds of MB per module, monorepos with many modules) need graphify-out out of git and shared through a central place. This PR adds that without touching the skill: a committed .graphify/config.json selects a store folder, graphify keeps every ./graphify-out as a link into it, and push/pull hooks committed in the repo sync the store anywhere (S3, rsync, git-lfs, a network share).

One command group:

graphify remote init      # scaffold .graphify/ (config.json + starter push/pull hooks) — commit it
graphify remote push      # run the push hook — upload the store tree
graphify remote pull      # run the pull hook — download the store tree + recreate all links
graphify remote delete    # leave the store — links become real local folders again

Why a link (and not path rewriting)

The skill, the CLI defaults, and users' habits all use the literal graphify-out/... path (75 references in skill.md alone). A link keeps every one of those working unchanged while the bytes live outside the repo — graphify/skill*.md, graphify/skills/, and tools/skillgen/ are byte-identical to v8 in this PR (--check: 134 artifacts OK). The filesystem does the redirection, so the CLI and the skill can never disagree.

Behaviour

  • Any graphify command ensures the link first (one lstat when already correct). No config → identical to today.
  • The link target is <store>/<module-relpath>/graphify-out — POSIX symlink, Windows directory junction (no admin rights).
  • The store path is the whole key — no <repo>/<branch> segment. The repo points at the same store location on every branch (switch branches → no retarget, no rebuild; graphify update refreshes). Each repo names its own store path in its config; remote init defaults it to ~/graphify-store/<folder>, editable to anything.
  • Monorepos: one link per directory you build in, keyed by its repo-relative path.
  • Adoption is safe: a pre-existing real graphify-out/ — even one tracked in git — is migrated into the store once; committing then records the graph files leaving git.
  • Exit is safe: remote delete replaces each link with a real folder holding a copy of its store data (the store is untouched), so a team can adopt and later go back. Both directions verified live.
  • .gitignore gets a bare graphify-out entry in the same step (bare on purpose: graphify-out/ matches only directories — git sees a symlink as a file, and on Windows git would descend into an unignored junction). Even unignored, a symlink commits as a ~100-byte path, never data.
  • A regular file named graphify-out is never clobbered (warns and skips).
  • GRAPHIFY_OUT (absolute) still overrides everything and disables linking.

Sync hooks

remote push/pull run a hook — Python, Node, shell, PowerShell, .cmd/.bat, or any executable. Resolution: config "push"/"pull" path → .graphify/<action>.* committed in the repo. Repo-local hooks mean a clone carries its own sync behaviour — zero per-machine setup; secrets stay in the environment (~/.aws, env vars), never in the repo.

The hook gets GRAPHIFY_STORE_DIR (the <store> tree), GRAPHIFY_PREFIX (the store folder basename), GRAPHIFY_CONFIG, GRAPHIFY_STORE, GRAPHIFY_REPO_ROOT, and the action in the env and mirrors that tree; graphify has no backend dependency. An executable hook runs directly (shebang wins); otherwise the interpreter is picked by extension, so it also works on Windows.

remote init scaffolds working S3/MinIO starters (boto3 via uv shebang; cache/ excluded as a local-only accelerator) with header comments telling teams how to swap languages. remote pull also fans a link out for every module found in the store, so one pull on a fresh clone recreates the whole working set. If the store folder is already shared (NFS/Dropbox), push/pull are unnecessary.

Files

  • graphify/store.py (new) — config discovery, store keying, link/junction creation, migration, materialization, gitignore upkeep
  • graphify/remote.py (new) — the graphify remote <init|push|pull|delete> group: hook resolution + invocation, bootstrap, exit
  • graphify/remote_hook_templates.py (new) — S3 starter hooks
  • graphify/__main__.py / graphify/cli.py — dispatch + help
  • docs/graph-store.md (follows the existing docs/ feature-doc convention), tests/test_remote_sync.py (49 offline unit tests)

Testing

  • pytest tests/ → 3049 passed (Python 3.12), python -m tools.skillgen --check → OK, ruff/pyright clean.
  • The 48 unit tests cover the full matrix: config discovery edge cases (malformed/non-dict JSON, non-string store, ~ expansion, the store path as the sole key), linking (idempotency, branch-independence, dangling-link self-heal, custom GRAPHIFY_OUT names, file-collision safety), migration (including collisions — local wins), gitignore variants (bare/slash/**, missing trailing newline), hook resolution (auto-discovery without a config key, extension priority, all six interpreters, executable/shebang path), the full hook env contract, remote dispatch incl. the deinit alias, init bootstrap/idempotency/other-language respect, materialization, and the S3 template's cache/ exclusion executed offline.
  • Live end-to-end on macOS: a real 2300-node/5200-edge production monorepo (query, path, explain, benchmark, callflow-HTML dashboard all read/write through the link; repo holds 0 bytes) plus a synthetic monorepo sandbox → builds land physically in the store through the links; git status clean; branch switch keeps the same link (branch-independent, no rebuild); publisher remote push + fresh-clone teammate remote pull verified against a real S3 bucket (Hetzner Object Storage) with the scaffolded boto3 hooks (6 objects round-tripped under one flat <store> prefix, cache excluded). Adoption + exit scenario verified live with graphs previously committed to git (2 tracked files → 0 → 2).

Closes #1751

@muthuishere muthuishere force-pushed the feat/config-store-and-sync-hooks branch 2 times, most recently from 31729da to 45f075f Compare July 9, 2026 21:24
@muthuishere muthuishere marked this pull request as draft July 10, 2026 02:18
@muthuishere muthuishere force-pushed the feat/config-store-and-sync-hooks branch from 45f075f to 3a7de52 Compare July 10, 2026 02:30
@muthuishere muthuishere changed the title feat: configurable graph store folder + pluggable push/pull sync hooks feat: central graph store (graphify-out links into a store folder) + pluggable push/pull sync hooks Jul 10, 2026
@muthuishere muthuishere force-pushed the feat/config-store-and-sync-hooks branch 2 times, most recently from bcbdc06 to caf0e67 Compare July 10, 2026 02:55
@muthuishere muthuishere force-pushed the feat/config-store-and-sync-hooks branch 2 times, most recently from 24f3ad1 to 8390b30 Compare July 10, 2026 03:26
@muthuishere muthuishere changed the title feat: central graph store (graphify-out links into a store folder) + pluggable push/pull sync hooks feat: central graph store (graphify remote init/push/pull/delete) — graphify-out links into a store folder + repo-local sync hooks Jul 10, 2026
…repo-local sync hooks (graphify remote)

A committed .graphify/config.json ({"store": "~/graphify-store"}) makes every
graphify run keep ./graphify-out as a link (POSIX symlink / Windows directory
junction, no admin) into <store>/<repo>/<branch>/<module-relpath>/graphify-out.
Writes physically land outside the repo; every literal graphify-out/... path —
CLI defaults, the agent skill, plain cat — keeps working through the link, so
the skill and skillgen artifacts stay byte-identical to upstream.

One command group: graphify remote <init|push|pull|delete>
- remote init: bootstrap the committed .graphify/ (config.json + starter S3
  push/pull hooks; swap for .sh/.js/.ps1/.cmd of the same name or point config
  {"push": ...} anywhere; creds stay in env, never in the repo)
- remote push/pull: run the repo's hook with GRAPHIFY_STORE_DIR/PREFIX/CONFIG
  in the env (resolution: config path -> .graphify/<action>.* ->
  ~/.config/graphify/hooks/); pull then fans a link out for every module in
  the store — one pull recreates a fresh clone's whole working set
- remote delete: leave the store — every link becomes a real local folder
  holding a copy of its store data (store untouched)

Adoption/exit are both safe: pre-existing local graphify-out dirs (even ones
tracked in git) migrate into the store once; a bare 'graphify-out' .gitignore
entry is maintained (trailing-slash misses symlinks; bare also stops git
descending junctions on Windows). Repo key from the origin remote basename so
all clones share it; branch switches retarget the link. No config -> behaviour
identical to upstream.
@muthuishere muthuishere force-pushed the feat/config-store-and-sync-hooks branch from 8390b30 to c8b1268 Compare July 10, 2026 03:37
… segments

The store path is now the whole key: graphify-out links into
<store>/<module>/graphify-out with no <repo>/<branch> segments. A repo points at
the same store location on every branch (no retarget, no rebuild; `graphify
update` refreshes), and each repo just names its own store path — `remote init`
defaults it to ~/graphify-store/<folder>. Removes the origin-URL parsing and
branch derivation.

Hook env: GRAPHIFY_STORE_DIR is the <store> tree, GRAPHIFY_PREFIX the store
basename; GRAPHIFY_REPO/GRAPHIFY_BRANCH removed. Docs + 48 tests updated.
@muthuishere muthuishere marked this pull request as ready for review July 10, 2026 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Central graph store: keep graphify-out out of git, share via repo-local push/pull hooks (graphify remote)

1 participant